Obtaining physically compatible reduced order models between scales remains as a great challenge in computational mechanics. This is of great importance in problems with several time and length scales, such as molecular dynamics, which are very difficult to completely resolve all at once. The basic framework to tackle this problem is based on the theory of coarse graining in statistical mechanics. In particular, the Mori-Zwanzig formalism describes how by coarse-graining a system, both viscous (memory) and stochastic effects arise. Some of the most successful methods in the field follow this principle, such as dissipative particle dynamics [1] and its smoothed version [2]. In this work, we develop a deep learning methodology to obtain coarse-grained models from data and its evolution in time. The model is forced to satisfy the laws of thermodynamics by construction, and is able to reproduce the statistical properties of the original system. The problem is formulated as a meshfree method and is not constrained to any particular system, being applicable to both micro, meso or macroscopic phenomena. The learning procedure is also scalable to large-scale dynamical systems.